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Abstract 

This paper reports the results of the Special Study Survey completed by 370 AACTE 
membership institutions in conjunction with the Professional Education Data System (PEDS) 
survey in fall of 2001. The purpose of the survey was to identify and describe what schools, 
colleges, and departments (SCDEs) use to measure candidate outcomes. The survey also solicited 
information regarding whether the measures used by SCDEs are mandated by the state or unit, 
whether the measures are tied to certification and licensure, the source of the measures (e.g., 
commercially produced or developed by the SCDE), the knowledge bases for the outcomes 
measures, the frequency with which data from the outcomes measures are reported, to whom the 
results are reported, and how the data from the outcomes measures are used to inform programs. 

The Special Study yielded promising information regarding the extent to which schools, 
colleges, and departments of education are responding to more rigorous standards and to national 
and state mandates for accountability. Through multiple types of outcomes measures, the majority 
of teacher education institutions responding to the survey are collecting data relative to candidate 
knowledge, skills, and dispositions. 
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Teacher Education Outcomes Measures: 

Special Study Survey 

In the past 10 years teacher quality has been identified as the single most important factor 
accounting for the variability in student performance. A growing body of research has established 
the quality of the teacher as the largest variable in student learning, even when students’ prior 
education, socioeconomic status, and other factors are included (Ferguson, 1991; Ferguson & Ladd, 
1996; Greenwald, Hedges, & Laine, 1996; Wright, Horn, & Sanders, 1997). Emerging from this 
confirmation of the critical role of quality teachers are numerous national and state mandates for 
holding teacher education institutions accountable for the preparation of educators who possess the 
knowledge, skills, and dispositions to support the learning and well-being of all students. At the 
heart of these mandates is the assertion that the preparation of teacher candidates must include 
understanding of content standards for students and what it takes to enable students to achieve them 
(Darling-Hammond, 2000). The expectation is that candidates should know their subject matter, 
demonstrate the ability to teach it to diverse groups of students, and be able to assess student 
learning effectively. 

Federal and state policymakers have employed a variety of mechanisms to ensure teacher 
candidates have subject-matter competency and the skills to help students master content. Some 
mechanisms, such as raising admission standards and testing new teachers, put the responsibility 
for mastery on prospective teachers. Others hold the teacher preparation program responsible. 
Taking the latter approach, the National Council for the Accreditation of Teacher Education 
(NCATE, 2000) has developed performance-based evaluation standards that require teacher 
education institutions to provide evidence of candidate knowledge, skills, and dispositions and the 
impact of their candidates and graduates on student learning. 

Prompted by new federal Title II legislation requiring all states to rank their teacher 
preparation institutions, states are also developing accountability criteria. These accountability 
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criteria, however, focus almost exclusively on the assessment of teacher content knowledge via 
standardized subject matter assessments. While subject matter knowledge is certainly an important 
aspect of teacher quality, teachers who support student learning also demonstrate the ability to use a 
broad repertoire of instructional skills and strategies that enable the learning of all students. 
Moreover, effective teachers are able to assess student learning and use assessment data to inform 
and improve the learning and teaching process (Education Commission of the States, 2000). The 
challenge faced by teacher education institutions is to provide evidence that their candidates and 
graduates possess the knowledge, skills, and dispositions to support the learning and well-being of 
all students. 

Anticipating the mandates for teacher education accountability, the American Association of 
Colleges for Teacher Education (AACTE) identified the development of comprehensive assessment 
systems as a priority for its membership institutions. In setting this priority, AACTE 
commissioned Idaho State University to conduct a national survey of teacher education institutions 
to identify and describe what schools, colleges, and departments of education (SCDEs) use to 
measure candidate outcomes. Through the survey, we sought to identify the types of outcomes 
measures used by SCDEs; whether the measures are mandated by the state and tied to certification 
or licensure; the source of the measures (e.g., commercially produced or developed by the SCDE); 
the knowledge base for the outcomes measures; the frequency with which data from the outcomes 
measures are reported and to whom the results are reported; and how data from the outcomes 
measures are used to inform programs. 

, Methods 

Participants 

The Special Study Survey was accessible by approximately 750 AACTE member 
institutions who were asked to complete the special survey on the Internet along with their annual 
Professional Education Data System (PEDS) survey. Data were received from N = 370 of the 
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schools, colleges, and departments of education (SCDE), for an approximate return rate of 50% of 
the membership institutions. Table 1 presents the frequency and percent by position of the persons 
responding to the survey. As can be seen from Table 1 , the respondents held a variety of positions. 
Among the most frequent responders were deans (20.3%), associate deans (15.9%), and department 
chairs or division directors (21 .4%). Of the 370 responding institutions, n = 363 (98. 1 %) indicated 
their SCDE collected data on candidate outcomes and completed the remainder of the survey. 

Seven of the institutions (1.9%) indicated their SCDE did not collect data on candidate outcomes, 
and exited the survey as instructed without responding to the remainder of the items on the 
questionnaire. Of the 363 institutions who collected data on candidate outcomes, the responses for 
/= 343 (94.5%) were based on their elementary teacher preparation program only,/= 8 (2.2%) 
were based on their early childhood teacher preparation program only, and /= 5 (1.4%) were based 
on their secondary teacher preparation program only. Four SCDEs (1.1%) said the responses were 
for all of there K-12 teacher preparation programs, even though the instructions asked them to 
respond for their largest program only. The information for program type was missing for/= 3 
(0.8%) of the institutions collecting candidate outcomes data. 



Insert Table 1 About Here 



Instrument 

The purpose of the survey was to identify and describe the kinds of assessments used by 
schools, colleges, and departments of education to measure outcomes for candidates in their 
programs for the initial preparation of teachers. Candidate outcomes were defined as the 
knowledge, dispositions, and skills the SCDE expects candidates to develop as a result of the 
teacher preparation program. The survey was a self-administered electronic factual questionnaire. 
Respondents were first asked to submit information about the name of the institution, its AACTE 
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identification number, and the title of the person completing the survey. The first item was a yes or 
no question about whether the school, college, or department of education (SCDE) collected data on 
candidate outcomes. Respondents who checked yes were instructed to complete the remainder of 
the questionnaire. 

The first part of the questionnaire consisted of five items that solicited factual information 
about the program type, the knowledge base used for the candidate outcomes measures, the uses of 
the data from the measures, the audiences for the data, and how frequently reports of the data are 
made. These questions consisted of multiple response checklists. The second part of the 
questionnaire identified measures of candidate outcomes. For each measure used by the SCDE, the 
respondents were asked to check all boxes that applied as to whether the measure is mandated and 
by whom; whether the measure is required for certification; and whether the source of the measure 
is commercial, state developed, or developed by the unit. 

The Special Study Survey questionnaire was constructed by the authors and approved by 
the AACTE Research and Information Committee. The questionnaire was placed on the AACTE 
web site by the Research and Information Services Division of AACTE. Pilot testing of the 
electronic format of the survey was conducted by the Research and Services Division with 5 
institutions randomly selected from the AACTE membership institutions. Appendix A contains a 
copy of the entire questionnaire. The Special Study Survey continues to be available on the 
AACTE web site (www.AACTE.org). The data for this study were collected from August 1, 2001 
until December 15, 2001. 

Procedures 

The Special Study Survey was activated at the AACTE web site on August 1, 2001. 
Authorized persons at member institutions completed the self-administered electronic survey. All 
institutions were expected to respond by the September 15, 2001 deadline for submission of the 
PEDS survey. When completed forms were submitted, the responses were attached to an e-mail 
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message sent to Idaho State University on a daily basis by the AACTE Research and Services 
Division. Responses to the questionnaire were accepted until December 15, 2001 (even though the 
deadline had passed three months earlier). 

Limitations 

A limitation of the Special Study Survey was a lack of formal procedures by AACTE for 
follow-up with the institutions that did not respond to the PEDS survey. Due to the modest return 
rate (only about 50% of the member institutions) and the fact the institutions who are not collecting 
outcomes data on their candidates would have a bias to not respond, the findings of this survey are 
limited to the institutions who actually responded. A list of the institutions that responded to the 
Special Study Survey is attached as Appendix B. 

An additional limitation to the findings of this study was caused by the electronic format of 
the survey. A major difficulty was caused by the initial data structure created by the AACTE 
Research and Information Services and the amount of information it accepted for each case. The 
amount of information entered for the open-ended response items often caused the responses for 
one or more of the last four outcome measures to be lost, because the case data were simply 
truncated. Eventually, this problem was recognized and fixed. However, the majority of the 
responses to the survey had already been submitted by that time. As a consequence, the findings 
for the last four outcomes measures are limited by the missing data. For the last two items, these 
limitations are severe. 

Results 

Knowledge Base for Candidate Outcomes Measures 

The knowledge base for the SCDEs’ candidate outcomes measures came from several sets 
of standards. NCATE Unit Accreditation Standards were used by/= 309 or 85.1% of the SCDEs. 
State standards were used by/= 345 or 95% of the institutions that collect data on candidate 
outcomes. National standards were employed as a knowledge base for candidate outcomes by 
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/= 250 or 68.9% of the institutions collecting outcomes data. Of those SCDEs using national 
standards, the commonly mentioned standards were the Interstate New Teacher Assessment and 
Support Consortium (INTASC), National Board for Professional Teaching Standards (NBPTS), 
Association for Childhood Education International (ACEI), International Reading Association 
(BRA), National Council of Teachers of Mathematics (NCTM), and the International Society for 
Technology in Education (ISTE). 

Uses of Data From Candidate Outcomes Measures 

According to the institutions responding to the survey, the data from the candidate outcomes 
measures are used for multiple purposes. Table 2 presents the frequency and percent of institutions 
using data from candidate outcomes by type of use. A high percentage of the institutions reported 
using the data for each of the purposes listed. The most frequent use was “to improve program 
effectiveness” (96.7%). This was followed closely by the use of candidate outcomes data “for 
accreditation documentation” (94.8%). For the/= 45 (12.4%) institutions indicating other uses, 
the regularly mentioned uses included “to provide candidates feedback for improvement,” “public 
relations,” “Title II reporting,” and “to advise students.” 



Insert Table 2 About Here 



Audiences for Data From Candidate Outcomes Measures 

Across institutions, the data from the candidate outcomes measures are released to multiple 
audiences. Table 3 presents the frequency and percent of institutions reporting data from candidate 
outcomes measures by audience type. The most frequent audience was unit and program faculty 
(95.3%), followed by State Departments of Education (91.2%), and Title II Higher Education Act 
reporting (87.6%). Only 55.9% of the institutions collecting candidate outcomes data release the 
data to the public and only 43.3% release the data to the media. A small percentage (8.8%) of the 




9 



Special Study Survey 9 



institutions cited additional audiences for their data. Other audiences to whom data are released 
included “stakeholder groups,” “alumnae,” “prospective students,” and “K-12 public schools.” 



Insert Table 3 About Here 



Frequency of Reports of Data From Candidate Outcomes Measures 

The data from candidate outcomes measures are reported regularly with different measures 
being reported at different times and for different purposes. Across measures, /= 140 institutions 
(38.6%) said they reported data from candidate outcomes measures each semester, /= 295 (81 .3%) 
said they reported data “annually, ”/= 81 (22.3%) said “every five years, ”/= 147 (40.5%) said 
“when data become available.” Some institutions (f= 24; 6.6%) said the data from the outcomes 
measures were reported at other times. The other times listed included “upon request,” 
“quarterly,” “at check points in the unit assessment system,” “every three years,” “every 10 
years,” and “frequency depends on the type of data.” 

Measures of Candidate Outcomes 

Table 4 presents the frequency and percent of institutions using each type of outcome 
measure for the schools, colleges, and departments of education (n = 363) that reported collecting 
candidate outcomes data for their initial elementary teacher preparation program or their largest 
teacher preparation program. Because this information was missing for some measures for some 
institutions from the data sets we received, the number of institutions out of the 363 for whom we 
we had useable data is also reported in Table 4. Percentages are based on the number of 
institutions out of the 363 for whom we had data for each measure. 



Insert Table 4 About Here 
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Tests of basic skills. Of the n = 363 institutions who reported collecting data from 
candidate outcomes measures, /= 349 or 96. 1 % reported using tests of basic skills in mathematics, 
reading, and/or writing as an outcome measure. For the institutions using tests of basic skills, 

/= 308 (88.3.%) reported these tests are state mandated, /= 215 (61.6%) reported these tests are 
mandated by the unit or SCDE, and/= 6 institutions (1.7%) said these tests are not mandated. For 
the tests of basic skills, /= 293 (83.9%) said successful performance on these tests is required for 
certification or licensure; while/= 44 (12.6%) said these tests are not required for certification or 
licensure. For/= 1 1 institutions (3.4%) this information was missing. The tests were reported to 
be commercially produced by /= 285 SCDEs (81.7%) and developed by the state by/= 59 
(16.9%) of the SCDEs. Only/= 24 SCDEs reported these tests were developed by the unit or 
SCDE. Because some institutions reported their tests were developed by more than one source, the 
total for these frequencies will exceeded the number of institutions using tests of basic skills as a 
measure of candidate outcomes. 

Interviews. Interviews are used for collecting data on candidate outcomes by/= 213 
(58.7%) of the SCDE. The number of SCDEs who said they are not using interviews was f= 148 
(40.8%). This information was missing for/= 2 (0.6%) institutions. Of the 213 SCDEs using 
interviews, /= 23 (10.8%) said they are state mandated, n = 184 (86.4%) said they are mandated by 
the unit or SCDE, and/= 25 (1 1.7%) said they are not mandated. Because the interviews are often 
mandated by both the state and the unit, these frequencies when added exceed the total number of 
institutions using interviews. Significantly, /= 84 (39.4%) of the 213 institutions using interviews 
as a measure of candidate outcomes said the interviews are required for certification or licensure. 
The number reporting they are not required for certification or licensure was /= 106 (49.8%). This 
information was missing for/= 23 (10.8%) of the institutions. Overwhelmingly, the institutions 
( f= 204; 95.8%) reported the interviews were developed by the unit or SCDE. One institution 
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reported the interviews were commercially produced and one institution reported they were 
developed by the state. 

Tests of subject matter knowledge. Tests of subject matter knowledge (such as Praxis II) 
are required by/= 306 (84.3%) of the n = 363 institutions collecting data on candidate outcomes. 
The number of SCDEs not using tests of subject matter knowledge was /= 57 (15.7%). These 
tests were reported as state mandated by /= 284 (92.8%) of the 306 institutions . They were 
reported as unit mandated by /= 137 (44.8%) and as not mandated by/= 1 1 (3.6%). Again, some 
SCDEs reported these tests are mandated by both the state and the unit. The tests of subject 
knowledge were reported as required for certification or licensure by/= 278 (90.8%) of the 
institutions collecting candidate outcomes data. Another/= 18 (5.9%) institutions reported the 
tests of subject matter knowledge are not required for certification or licensure. For/= 10 
institutions (3.3%), this information was missing. Most SCDEs (/’= 256; 83.7%) stated the tests of 
subject matter knowledge were developed commercially. Some SCDEs (f= 46; 15.0%) said they 
were developed by the state. A few SCDEs (f= 16; 5.2%) said they were developed by the unit. 

Portfolios. Portfolios are used as a measure of candidate outcomes by/= 319 (87.9%) of 
the institutions collecting data on candidate outcomes. Of the 319 institutions using portfolios, 

/= 64 (20.1%) said they are mandated by the state, /= 269 (84.3%) said they are mandated by the 
unit, and /= 37 (11 .6%) said they are not mandated. Portfolios were reported as required for 
certification or licensure by/= 123 (38.6%) of the SCDEs and as not required for certification or 
licensure by/= 159 (49.8%) of the SCDEs. This information was missing for/= 37 (1 1.6%) of 
the institutions that reported using portfolios as an outcomes measure. None of the institutions 
reported using commercially developed portfolios. Only a few of the institutions (f= 22; 6.9%) 
said the portfolios were developed by the state. Most of the institutions (f= 305; 95.6%) said the 
portfolios were developed by the unit. 
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Attitude surveys/self-ejficacy scales. Attitude surveys and self-efficacy scales (such as the 
Teacher Perceiver) were reported as measures being used to assess candidate outcomes by/= 108 
(29.8%) of the n = 363 institutions collecting outcomes data. These measures were reported as 
state mandated by /= 9 SCDEs (8.3%), as unit or SCDE mandated by /= 75 (69.4%), and as not 
mandated by/= 28 (25.9%) of the institutions using these measures. Again, due to the overlap of 
state and unit mandates, these frequencies do not sum to the total number of institutions using 
attitude surveys or self-efficacy measures. Only/= 26 institutions (24.1%) said the attitude 
surveys or self-efficacy scales are required for certification or licensure; whereas, /= 65 institutions 
(60.2%) said they are not required for certification or licensure. This information was missing for 
/=17 (15.7%) of the institutions who reported using attitude surveys or self-efficacy scales as 
outcomes measures. In general, the attitude surveys or self-efficacy scales were reported to be 
developed by the Unit or SCDE (f= 93; 86. 1 %). These measures were reported to be 
commercially produced by only /= 13 of the SCDE (12.0%) and developed by the state by only 
/= 4 (3.7%) of the SCDE. Some institutions reported these measures were developed by more 
than one source, so the total numbers do not add to the number of institutions using these measures 
to collect outcomes data on their candidates. 

Teacher work samples. Teacher work samples are used as a measure of candidate 
outcomes by/= 241 institutions (66.4%). Forty-five (18.7%) of the 241 SCDEs reported teacher 
work samples are mandated by the state. Another/= 190 SCDEs (78.8%) said teacher work 
samples are mandated by the unit or SCDE. Thirty-four (14.1%) of the 241 SCDEs using teacher 
work samples said they are not mandated. Again, these numbers reflect the fact that some 
institutions reported both state and unit mandates. The number of SCDE (out of the 241) who 
reported that teacher work samples are required for certification or licensure was /= 85 (35.3%). 
Oddly, more institutions said they are required for certification or licensure than said they are 
mandated by the state. The number of SCDE who reported they are not required for certification or 
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licensure was /= 135 (56%). This information was missing for/= 21 (8.7%) institutions who said 
they used teacher work samples as an outcomes measure. The vast majority, /= 228 (94.6%), of 
the 241 SCDEs who used teacher work samples said they were developed by the unit. Only/= 2 
(0.8%) SCDEs reported they used commercially produced teacher work samples, and only /= 1 1 
(4.6%) said they were developed by the state. 

Measures of PK-12 student learning. Of the n = 363 institutions who reported using 
candidate outcomes measures, only /= 103 (28.4%) reported using measures of P-12 student 
learning as one of their assessments. For the 103 SCDEs using measures of P-12 student learning, 
/= 25 (24.3%) said it is because they are state mandated. The majority (f= 73; 70.9%) of the 
SCDEs said the measures of P-12 student learning are mandated by the unit or SCDE. Some 
institutions (f= 18; 17.5%) said they are not mandated. As stated previously, these figures reflect 
the fact that some institutions reported both state and unit mandates. Out of the 103 institutions 
using measures of P-12 student learning as measures of candidate outcomes, /= 33 (32%) said the 
measures are required for certification or licensure, and/= 58 (56.3%) said they are not required. 
This information was missing for /= 12 institutions (1 1.7%). More of the SCDEs (f= 76; 73.8%) 
reported the measures of P-12 student learning were developed by the unit or SCDE, than reported 
they were developed commercially (f= 14; 13.6%) or reported they were developed by the state 
( f= 13; 12.6%). 

Case study analyses/problem-solving scenarios. Case study analyses or problem-solving 
scenarios were reported as measures being used to assess candidate outcomes by/= 168 (46.3%) 
of the n = 363 institutions collecting outcomes data. These measures were reported as state 
mandated by/= 9 SCDEs (5.4%), as unit or SCDE mandated by/= 141 (83.9%), and as not 
mandated by/= 27 (16.1%) of the institutions using these measures. As was true for the previous 
measures, these figures reflect the fact that some institutions reported both state and unit mandates. 
Only /= 42 institutions (25.0%) said the case study analyses or problem-solving scenarios are 
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required for certification or licensure; whereas, /= 108 institutions (64.3%) said they are not 
required for certification or licensure. This information was missing for /=18 (10.7%) of the 
institutions who reported using case study analyses or problem-solving scenarios as outcomes 
measures. As might be expected, most of the case study analyses or problem-solving scenarios 
were reported to be developed by the unit or SCDE (n = 157; 93.5%). These measures w ere 
reported to be commercially produced by only/= 10 of the SCDEs (6.0%) and developed by the 
state by only/= 2 (1.2%) of the SCDEs. Some institutions reported these measures were 
developed by more than one source. 

Teaching artifacts. Teaching artifacts, such as lesson plans or unit plans, are used as a 
measure of candidate outcomes by /= 332 (91.5%) of the institutions collecting data on candidate 
outcomes. Of the 332 institutions using teaching artifacts, /= 52 (15.7%) said they are mandated 
by the state, /= 289 (87.0%) said they are mandated by the unit, and /= 22 (6.6%) said they are not 
mandated. Some institutions reported both state and unit mandates, so these figures exceed the total 
number of institutions using teaching artifacts. Teaching artifacts were reported as required for 
certification or licensure by /= 128 (38.6%) of the SCDEs and as not required for certification or 
licensure by /= 172 (51.8%) of the SCDEs. This information was missing for/= 32 (9.6%) of the 
institutions that reported using teaching artifacts as a candidate outcomes measure. As might be 
anticipated, most of the institutions (f= 306; 92.2%) said the teaching artifacts were developed by 
the unit. Only one institution reported using commercially developed teaching artifacts, and very 
few of the institutions (f= 12; 3.6%) said the teaching artifacts were developed by the state. 

Tests of professional knowledge. Of the n = 363 institutions who reported collecting data 
on candidate outcomes, /= 200 or 55.1% reported using tests of professional knowledge, such as 
the National Teacher Examinations, as a candidate outcomes measure. For those 200 institutions 
using tests of professional knowledge, /= 180 (90.0%) reported these tests are state mandated, 

/= 93 (46.5%) reported they are mandated by the unit or SCDE, and only /= 10 institutions (5.0%) 
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said these tests are not mandated. Many of the SCDEs said the tests are mandated by both the state 
and the unit. For these tests of professional knowledge, /= 170 of the SCDEs (85.0%) said 
successful performance is required for certification or licensure; while/= 15 (7.5%) said these tests 
are not required for certification or licensure. For /= 15 institutions (7.5%) this information was 
not reported. The tests of professional knowledge were reported to be commercially produced by 
/= 157 of the SCDEs (78.5%) and developed by the state by/= 28 (14.0%) of the SCDEs. Only 
/= 21 of the SCDEs (10.5%) reported these tests were developed by the unit or SCDE. Because a 
few institutions reported the tests were developed by more than one source, the total for these 
figures exceeds the number of institutions using tests of professional knowledge as a measure of 
candidate outcomes. 

Teaching performance evaluations. Teaching performance evaluations from direct 
observations or from video are used as a measure of candidate outcomes by/= 339 (93.4%) of the 
363 SCDEs collecting candidate outcomes data. Only/= 92 (27.1%) of the 339 SCDEs using 
teaching performance evaluations said they are mandated by the state. Most SCDEs (f= 300; 
88.5%) said they are mandated by the unit or SCDE. Six institutions (1.8%) said they are not 
mandated. Again, these numbers reflect the fact that some institutions reported both State and Unit 
mandates. The number of SCDEs (out of the 339) who reported that teaching performance 
evaluations are required for certification or licensure was /= 174 (5 1 .3%). The number of SCDEs 
who reported they are not required for certification or licensure was /= 135 (39.8%). This 
information was not reported for/= 30 (8.8%) of the institutions who reported using teaching 
performance evaluations as an outcomes measure. The vast majority of the SCDEs (f= 304; 

89.7%) said their teaching performance evaluations were developed by the unit. Only/=5 SCDEs 
(1.5%) reported they used commercially produced teaching performance evaluations, and only 
/= 35 (10.3%) said they were developed by the state. 
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Follow-up surveys with program completers. Follow-up surveys of program completers 
were reported as a measure of candidate outcomes by/= 307 of the 363 institutions (84.6%) 
collecting data on candidate outcomes. The surveys were reported as mandated by the state by 
/= 88 of the institutions (28.7%) and as mandated by the unit or SCDE by/= 248 (80.8%) of the 
institutions. They were reported as not mandated by /= 33 (10.7%) of the SCDEs. Most 
institutions (f= 219; 71.3%) reported these measures are not required for certification or licensure. 
Surprisingly,/= 43 institutions (14.0%) said that surveys of program completers are required for 
certification or licensure. This information was missing for/= 45 (14.7%) of the 307 institutions 
who reported using surveys of program completers. Most of the surveys (f = 263; 85.7%) were 
developed by the unit or SCDE. Some institutions, /= 35 (1 1.4%), said they were developed by the 
state. Only /= 12 (3.9%) of the institutions using surveys of program completers said they were 
developed commercially. 

Candidate journals. Information about the use of candidate journals was missing for/= 55 
of the 363 institutions who reported collecting data on candidate outcomes. For the 177 SCDEs 
using candidate journals, /= 147 (83. 1%) said they are mandated by the unit or SCDE and/= 25 
(14.1%) said they are not mandated. Only /= 8 (4.5%) reported they are mandated by the state. 
This information was missing for a few institutions. Interestingly, /= 30 of the 177 SCDEs 
(16.9%) using candidate journals reported they are required for certification or licensure. However, 
most SCDEs (f= 102; 57.6%) said they are not required for certification or licensure. This 
information was missing for /= 45 of the 177 institutions (25.4%) using candidate journals. None 
of the institutions using candidate journals reported they were developed commercially, and none of 
them said they were developed by the State (but this information was missing for/= 34 (19.2%) 
and /= 45 (35.4%) of the institutions respectively). As would be expected, most SCDEs (f= 116; 
65.5%) said the candidate journals were developed by the unit. This information was missing for 
/= 51 or 28.8% of the institutions. 
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Assessment of technology competency. Useable data on how many institutions employed 
assessments of technology competency as a measure of candidate outcomes was obtained for only 
/= 164 of the 363 institutions (45.2%) that reported collecting candidate outcomes data. For/= 

199 institutions (54.8%), this information was missing in the data sets we received. Of the 164 
institutions for whom we had data,/= 110 (67. 1 %) reported they used an assessment of technology 
competency as a measure of candidate outcomes, while /= 54 (32.9%) said they did not. The 
assessment of technology competency was reported as state mandated by/= 41 (37.3%) SCDEs, 
as unit or SCDE mandated by/= 77 (70.0%), and as not mandated by /= 8 (7.3%) of the SCDEs. 
For some institutions this information was missing, and as before some institutions reported both 
state and unit mandates. Forty -four institutions (40.0% of 1 10) reported an assessment of 
technology competency is required for certification or licensure. Twenty-eight (25.5%) institutions 
said it is not required for certification or licensure. This information was missing for /= 22 
(20.0%) of the 1 10 institutions that reported using an assessment of technology competency as a 
candidate outcomes measure. 

Tests of pedagogical-content knowledge. Unfortunately, information about the use of tests 
of pedagogical-content knowledge (such as Praxis III) was missing from the data sets we received 
for 257 of the 363 institutions that reported collecting data on candidate outcomes. We received 
useable data for only 106 institutions. From the data we received, /= 40 (37.7%) of the SCDEs 
said they use tests of pedagogical-content knowledge as a measure of candidate outcomes, and 
/= 66 (62.3%) said they do not use these tests. For those institutions that reported using the tests, 
/= 31 of the 40 (77.5%) institutions said they are state mandated, /= 17 (42.5%) said they are 
mandated by the unit or SCDE, and /= 2 (5.0%) said they are not mandated. This information was 
missing for some institutions and some institutions said they were mandated by both the unit and 
the state. Twenty-six of the 40 (65%) SCDEs for whom we had data indicating they use tests of 
pedagogical-content knowledge reported that these tests are required for certification or licensure. 
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Eight of the 40 SCDEs (20.0%) reported they are not required for certification or licensure. This 
information was missing for /= 6 (15%) of the 40 institutions . Twenty-four of the 40 institutions 
(60.0%) said the tests of pedagogical-content knowledge were commercially developed, 10 (25.0%) 
said they were developed by the state, and 8 (20.0%) institutions said they were developed by the 
unit. Some institutions reported more than one source. This information was missing for three of 
the 40 institutions. 

Employer surveys. Data about the use of employer surveys was missing from the data sets 
we received for 265 (73.0%) of the 363 institutions that reported collecting data on candidate 
outcomes. Of the 98 institutions for whom we received data,/= 53 (54. 1%) reported using 
employer surveys and/= 45 (45.9%) reported they do not use them. Of the 53 SCDEs for whom 
we had data indicating use of employer surveys, /= 15 (28.3%) said they are state mandated, /= 46 
(86.8%) said they are mandated by the unit or SCDE, and/= 4 (7.5%) said they are not mandated. 
Surprisingly, /= 9 (17.0%) of the 53 institutions using employer surveys reported they are required 
for certification or licensure. Thirty-four (64.2%) institutions said they are not required for 
certification or licensure. This information was missing for/= 10 of the 53 (18.9%) institutions 
for whom we had data indicating the use of employer surveys. Two of the 53 (3.8%) institutions 
said the employer surveys were commercially developed and f = 6 (1 1 .3%) said they were 
developed by the state. We only had valid data for two institutions indicating that the employer 
surveys were developed by the unit. This information was missing for 5 1 of the 53 institutions for 
whom we had data indicating use of employer surveys. Presumably, most of the employer surveys 
were developed by the unit, but we lack the data to verify this assumption. 

Conclusions 

The Special Study Survey yielded data relative to the types of outcomes measures used by 
schools, colleges, and departments of education. In addition, the survey yielded information 
regarding whether the measures used by SCDEs are mandated by the state or unit and whether the 
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measures are tied to certification and licensure. The survey also yielded data relative to major 
characteristics of the outcomes measures including the source of the measures (e.g., commercially 
produced or developed by the SCDE) and the knowledge bases for the outcomes measures. 

Finally, the survey yielded information regarding the use of the outcomes measures such as the 
frequency with which data from the outcomes measures are reported, to whom the results are 
reported, and how the data from the outcomes measures are used to inform programs. 

What Outcomes Measures Are Used by SCDEs? 

Survey responses indicate schools, colleges, and departments of education are using 
multiple types of measures to assess candidate knowledge, skills, and dispositions. These measures 
include tests of basic skills, subject matter knowledge, professional knowledge, and pedagogical- 
content knowledge; interviews; portfolios; case study analyses and problem-solving scenarios; 
journals; teacher work samples; teaching artifacts; teaching performance evaluations; attitude 
surveys and self-efficacy scales; measures of PK-12 student learning; and surveys with alumni and 
employers. 

The most common candidate outcomes measures used by SCDEs include tests of basic 
skills, teaching artifacts, and teaching performance evaluations as reported by 90% of more of the 
institutions responding to the survey. Eighty percent or more of the SCDEs reported using tests of 
subject matter knowledge, portfolios, journals, and follow-up surveys with program completers. 

The measures used least by SCDEs are attitude surveys and self-efficacy scales, tests of 
pedagogical content knowledge, and measures of PK-12 student learning. 

Results of the survey support the conclusion that a majority of schools, colleges, and 
departments of education are assessing candidate knowledge, skills, and dispositions with multiple 
measures during the course of the teacher preparation program. As such, it appears that a large 
number of teacher education institutions are responding to federal, state, and accreditation 
requirements for providing evidence of candidate performance. Interestingly, review of the Title II 




20 



Special Study Survey 20 



state reports (see www.title2.org) indicates that teacher preparation program accountability data is 
almost exclusively limited to standardized assessments of subject matter knowledge. The failure to 
include data from the other types of assessments used by SCDEs raises the question of whether 
important information regarding candidate performance and program effectiveness is being 
systematically excluded from decision-making about program quality. 

Are the Outcomes Measures Mandated by the State and/or the Unit? 

Responses to the Special Study Survey regarding whether the outcomes measures used by 
the institutions are mandated by the state and/or the unit indicate that the majority of outcomes 
measures used by SCDEs are mandated by the unit. In addition, data from the survey indicate that 
the source of the mandate (e.g., state or unit) differ by the type of assessment. For example, tests of 
basic skills, professional knowledge, and subject matter knowledge tend to be mandated by the state 
while other measures such as interviews, portfolios, teaching artifacts, and teaching performance 
evaluations are mandated by the unit. 

As with the survey responses regarding the types of outcomes measures used by SCDEs, 
these data indicate that institutions are assessing candidate performance beyond the measures 
resulting from state mandates or Title II requirements. With the NCATE 2002 standards requiring 
units to use comprehensive assessment systems and to provide evidence of candidate performance, 
it is reasonable to hypothesize that the unit mandates for outcomes measures are a result of 
institutional efforts to meet new and more rigorous accreditation standards. 

Are the Outcomes Measures Tied to Certification or Licensure? 

Survey responses indicate that some of the outcomes measures used by schools, 
departments, and colleges of education are tied to certification or licensure. The measures most 
frequently tied to certification or licensure include tests of basic skills, subject matter knowledge, 
and pedagogical-content knowledge. About 84% of the institutions using tests of basic skills said 
the tests are tied to certification or licensure; 91% of the institutions using tests of subject matter 
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knowledge said the tests are required for certification or licensure; and 90% of the institutions using 
tests of professional knowledge said the tests are tied to certification or licensure. These figures are 
consistent with reports of the number of states currently requiring standardized tests of basic skills, 
subject matter knowledge, and professional knowledge for certification or licensure (Educational 
Testing Service, 2000). 

What Are the Sources of Outcomes Measures? 

Responses to the Special Study Survey regarding the sources (e.g., commercially produced 
or developed by the unit) of the outcomes measures indicate that the source depends on the type of 
measure. Tests of basic skills, subject matter knowledge, and professional knowledge tend to be 
commercially produced while all other outcomes measures are developed by the unit. These results 
support the inference that many institutions are committing significant resources and time to the 
development of candidate outcomes measures including interviews, teacher work samples, teaching 
performance evaluations, portfolios, case study analyses and problem solving scenarios, and alumni 
and employer surveys. 

What Are the Knowledge Bases of Outcomes Measures? 

Survey responses indicate that the knowledge bases for teacher education outcomes 
measures come from accreditation, national, and state standards. Of the institutions responding to 
the survey, 85% reported using the NCATE accreditation standards and 95% of the institutions said 
they used state standards as the basis for their outcomes measures. These figures provide evidence 
that schools, colleges, and departments of education are responding to the national and state 
mandates for accountability in teacher education by creating outcomes measures to assess candidate 
performance relative to state and professional standards for what teachers should know and be able 
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To Whom Are Data from Outcomes Measures Reported? 

Responses to the Special Study Survey regarding the audiences to whom SCDEs report 
data from outcomes measures indicate that institutions release candidate performance data to 
multiple audiences. The most commonly cited audience reported by responding institutions was 
unit and program faculty (95%), followed by State Departments of Education (92%), and Title II 
Higher Education reporting (88%). Only about half of the institutions collecting candidate 
outcomes data release the data to the public and only about 40% release the data to the media. 

These data support the conclusion that most schools, colleges, and departments of education are not 
taking advantage of the opportunity to release candidate performance data to policymakers and 
patrons. As noted by the Education Commission of the States (2000), in order to effectively 
respond to critics of teacher education, institutions must report data regarding the extent to which 
candidates and graduates are meeting state and national standards. 

How Frequently Are the Data from Outcomes Measures Reported? 

Survey responses indicate that schools, colleges, and departments of education frequently 
report data from outcomes measures. More than eighty percent of the institutions responding to the 
survey said they report candidate performance data annually, while nearly 40% said they report data 
from candidate outcomes measures each semester, and 40% said they report data when the data 
become available. These statistics indicate that institutions are reporting outcomes data more 
frequently than the one-year or five-year intervals required by the state or accreditation agencies. 

Are SCDEs Using Data from Outcomes Measures to Inform Programs? 

Responses from the Special Studies Survey indicate that schools, colleges, and departments 
of education use data from candidate outcomes measures for multiple purposes. The most 
frequently cited use of outcomes data cited by the responding institutions is to improve program 
effectiveness (96.7%). In addition, 95% of the responding institutions said they use candidate 
outcomes data for accreditation documentation. While data from the survey clearly establish that 
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SCDEs are using data from outcomes measures to inform programs, there is no information 
regarding precisely how the data is being used. A valuable follow-up study to the Special Study 
Survey would be to conduct e-mail surveys or interviews with a sample of the responding 
institutions to gather deeper information regarding the ways institutions are using data from 
outcomes measures to inform programs and to respond to national and state accountability 
mandates. 

Overall, the Special Study Survey yielded promising information regarding the extent to 
which schools, colleges, and departments of education are responding to more rigorous 
accreditation standards and to national and state mandates for accountability. Through multiple 
types of outcomes measures, the majority of teacher education institutions responding to the survey 
are collecting data relative to candidate knowledge, skills, and dispositions. However, teacher 
education institutions are increasingly expected to show program effectiveness in terms of the 
impact of their candidates and graduates on student learning (NCATE, 2000; Pankratz & Banker, 
2000). As such, SCDEs must expand their use of outcomes measures to include assessment 
approaches that connect teacher performance to PK-12 student learning. To participate as full 
partners in the improvement of teaching and learning in America’s schools, teacher preparation 
institutions must ground their programs in rigorous standards for what teachers should know and 
be able to do and use assessment systems that ensure their candidates and graduates support the 
learning and well-being of all students. 
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Table 1 



Frequency and Percent of Respondents by Position ( N = 370) 



Position 


f 


% 


Dean 


75 


20.3 


Associate Dean 


59 


15.9 


Assistant Dean 


18 


4.9 


Director of Teacher Education 


39 


10.5 


Department Chair/Division Director 


79 


21.4 


NC ATE/ Accreditation Coordinator 


16 


4.3 


Assistant/Associate Chair or Director 


4 


1.1 


Teacher Certification Officer 


5 


1.4 


Director of Institutional Research/Analyst 


8 


2.2 


Professor 


8 


2.2 


Secretary or Assistant to the Dean 


8 


2.2 


Other 


8 


2.2 


Missing 


43 


11.6 
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Table 2 

Frequency and Percent of Total ( n = 363) for Uses of Data From Candidate Outcomes Measures. 



Uses 


f 


% 


To make decision regarding candidate 
admission, retention, program completion 


342 


94.2% 


To recommend candidates for certification 
or licensure 


334 


92.0% 


For national and state accountability 
information 


327 


90.1% 


For accreditation documentation 


344 


94.8% 


To improve program effectiveness 


351 


96.7% 


Other purpose(s) 


45 


12.4% 
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Table 3 



Frequency and Percent of Total ( n = 363) of the Types of Audiences to Whom Data from 
Candidate Outcomes Measures Are Released. 



Audiences 


f 


% 


Regional accreditation agency 


218 


60.1% 


Program accrediting agency 


271 


74.7% 


Title II Higher Education Act Reporting 


318 


87.6% 


State Department of Education 


331 


91.2% 


Teacher Education Candidates 


256 


70.5% 


Unit/program faculty 


346 


95.3% 


Faculty in other units of the institution 


222 


61.2% 


University administration 


296 


81.5% 


Program/unit/institution advisory board 


255 


70.2% 


Public 


203 


55.9% 


Print and/or electronic media 


157 


43.3% 


Other audience(s) to whom data are released 


32 


8.8% 
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Table 4 



Frequency and percent of institutions using each type of outcomes measure based on the number of 
of schools, colleges, and departments of education for whom we had data indicating collection and 
use of candidate outcomes data. 



Outcomes Measure 


n 


f 


% 


Tests of Basic Skills 


363 


349 


96.1 


Interviews 


361 


213 


58.7 


Tests of Subject Matter Knowledge 


363 


306 


84.3 


Portfolios 


363 


319 


87.9 


Attitude Surveys/Seif-Efficacy Scales 


363 


108 


29.8 


Teacher Work Samples 


363 


241 


66.4 


Measures of PK-12 Student Learning 


363 


103 


28.4 


Case Study Analyses/Problem Solving Scenarios 


363 


168 


46.3 


Teaching Artifacts 


363 


332 


91.5 


Tests of Professional Knowledge 


363 


200 


55.1 


Teaching Performance Evaluations 


363 


339 


93.4 


Follow-Up Surveys with Program Completers 


363 


307 


84.6 


Candidate Journals 


308 


147 


83.1 


Assessment of Technology Competency 


199 


110 


67.1 


Tests of Pedagogical-Content Knowledge 


106 


40 


37.7 


Employer Surveys 


98 


53 


54.1 
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Appendix A 

,nslilulion: 1NST1D: Your Title: 

Schools, Colleges, and Departments of Education (SCDE) 

Teacher Education Outcomes Measures 

The purpose of this survey is to identify and describe what assessments schools, colleges, and departments of education use to measure outcomes 
for candidates in their programs for the initial preparation of teachers. Candidate outcomes are defined as the knowledge, dispositions, and 
skills the SCDE expects candidates to develop as a result of the teacher preparation program. 

PROGRAM TYPE - If your SCDE offers an elementary teacher preparation program, please respond for that program. If there is no elementary 
preparation program, please respond for the largest program you offer. 

The responses are for: 

Our elementary teacher preparation program. 

We have no elementary teacher preparation program. The responses are for 

Specify type of program 

KNOWLEDGE BASE FOR CANDIDATE OUTCOMES MEASURES — Please check all that apply 
The knowledge base for tbe SCDE’s candidate outcomes measures comes from: 

NCATE Unit Accreditation Standards 

State adopted standards 

Unit or program developed standards 

National standards: 

We do not collect candidate outcomes data Specify sourcefs): INTASC, NBPTS, etc. 

USE OF DATA FROM CANDIDATE OUTCOMES MEASURES — Please check all that apply 
Data from candidate outcomes measures are used: 

To make decisions regarding candidate admission, retention, program completion 

To recommend candidates for certification or licensure 

for national and state accountability information 

For accreditation documentation 

To improve program effectiveness 

Other purpose(s): 

We do not collect candidate outcomes data. Specify purpose(s) 

AUDIENCE FOR DATA FROM CANDIDATE OUTCOMES MEASURES - Please check all that apply 
Data from candidate outcomes measures are released to: 

Accreditation agency 

U S. Department of Education for Title 11 Reporting 

State Department of Education 

Teacher Education Candidates 

Unit/program faculty 

Faculty in other units of the institution 

University administration 

Program/unit/institution advisory board 

Public 

Print and/or electronic media 

Other audience(s) to whom data are released: 

We do not collect candidate outcomes data. Specify audiencefs) 

FREQUENCY OF REPORTS OF DATA FROM CANDIDATE OUTCOMES MEASURES - Please check all that apply 

Data from candidate outcomes measures are reported: 

Each semester 

Annually 

Every five years 

When data become available 

Other: 

We do not collect candidate outcomes data. 
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Appendix B 

SCDEs Responding to the Special Study Survey 



Adams State College 

Alabama A&M University 

Anderson College 

Andrews University 

Appalachian State University 

Arizona State University 

Arkansas Tech University 

Armstrong Atlantic State University 

Asbury College 

Ashland University 

Athens State University 

Atlanta Christian College 

Auburn University 

Auburn University Montgomery 

Augsburg College 

Augustana College 

Austin Peay State University 

Avila College 

Baker University 

Ball State University 

Bellarmine University 

Beloit College 

Benedictine College 

Berry College 

Bethany College 

Bethel College 

Bethune-Cookman College 

Binghamton University 

Black Hills State University 

Bloomsburg University of Pennsylvania 

Bluefield State College 

Boise State University 

Bowie State University 

Bowling Green State University 

Briar Cliff University 

California State University, Bakersfield 

California State University, Dominguez Hills 

California State University, Fullerton 

California State University, Hayward 

California State University, Long Beach 

California State University, Northridge 

California University of Pennsylvania 

Calvin College 

Campbell University 

Capital University 
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Cardinal Stritch University 
Carson-Newman College 
Catawba College 

Central Connecticut State University 
Central Washington University 
Clarion University of Pennsylvania 
Clark Atlanta University 
Clayton College & State University 
Clemson University 
Coe College 
College of Charleston 

College of Saint Benedict/ Saint John's University 

College of Saint Elizabeth 

College of the Ozarks 

Colorado College 

Colorado State University 

Columbia College 

Concord College 

Concordia University 

Concordia University (St Paul) 

Concordia University-Wisconsin 
Coppin State College 
Creighton University 
Culver-Stockton College 
Cumberland College 
Dakota State University 
Dana College 
DePaul University 
Delta State University 
Dickinson College 
Dickinson State University 
Doane College 
East Carolina University 
East Stroudsburg University 
Eastern Illinois University 
Eastern Michigan University 
Eastern Nazarene College 
Eastern New Mexico University 
Eastern Washington University 
Elizabeth City State University 
Elmhurst College 
Elon College 
Emporia State University 
Evangel University 
Fairmont State College 
Fitchburg State College 
Fontbonne College 
Fort Lewis College 
Francis Marion University 
Friends University 
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Frostburg State University 

Gardner-Webb University 

Georgia College & State University 

Georgia Southern University 

Georgian Court College 

Gonzaga University 

Governors State University 

Grand Canyon University 

Grand Valley State University 

Greensboro College 

Guilford College 

Gustavus Adolphus College 

Hamline University 

Harding University 

Harris-Stowe State College 

Hastings College 

Henderson State University 

Hendrix College 

Hofstra University 

Huntington College 

Idaho State University 

Illinois State University 

Indiana State University 

Indiana University East 

Indiana University School of Education 

Indiana University South Bend 

Indiana University Southeast 

Indiana University of Pennsylvania 

Indiana University-Purdue University 

Indiana Wesleyan University 

Jackson State University 

Jacksonville State University 

John Brown University 

Johns Hopkins University 

Kansas State University 

Keene State College 

Kennesaw State University 

Kentucky State University 

Kutztown University 

Lees-McRae College 

Lenoir-Rhyne College 

Lewis University 

Lewis-Clark State College 

Lindenwood University 

Livingstone College 

Lock Haven University 

Longwood College 

Louisiana State University and A&M College 
Louisiana State University- Shreveport 
Luther College 
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Lyon College 
Manchester College 
Marian College 

Marian College of Fond Du Lac 
Mars Hill College 
Marymount University 
Marywood University 
Mayville State University 
McNeese State University 
McPherson College 
Meredith College 
Methodist College 
Miami University 
Michigan State University 
Midland Lutheran College 
Millersville University of Pennsylvania 
Milligan College 
Millsaps College 

Minnesota State University Moorhead 
Minnesota State University, Mankato 
Mississippi College 
Mississippi State University 
Mississippi University for Women 
Mississippi Valley State University 
Missouri Baptist College 
Missouri Southern State College 
Montana State University 
Montana State University- Billings 
Montclair State University 
Montreat College 
Morehead State University 
Morgan State University 
Morningside College 
Murray State University 
Nebraska Wesleyan University 
New York University 
Niagara University 
Nicholls State University 
North Carolina A&T State University 
North Carolina Central University 
North Carolina State University 
North Carolina Wesleyan College 
North Dakota State University 
Northeastern State University 
Northern Kentucky University 
Northern State University 
Northwest College 
Northwest Missouri State University 
Northwest Nazarene University 
Northwestern College 
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Northwestern Oklahoma State University 

Northwestern State University 

Oakland City University 

Ohio Northern University 

Ohio University 

Ohio Wesleyan University 

Oklahoma Baptist University 

Oklahoma Christian University 

Oklahoma Panhandle State University 

Oklahoma State University 

Old Dominion University 

Otterbein College 

Pennsylvania State University 

Pfeiffer University 

Pittsburg State University 

Presbyterian College 

Providence College 

Purdue University 

Queens College 

Radford University 

Rhode Island College 

Rider University 

Rowan University 

Rutgers University 

Saint Cloud State University 

Saint Joseph's College 

Saint Louis University 

Saint Mary College 

Saint Xavier University 

Salem State College 

Salisbury State University 

Salve Regina University 

Sam Houston State University 

San Diego State University 

Seattle University 

Seton Hall University 

Shawnee State University 

Shepherd College 

Silver Lake College 

Simpson College 

Slippery Rock University of Pennsylvania 
South Carolina State University 
Southeastern Louisiana University 
Southern Adventist University 
Southern Illinois University Carbondale 
Southern Illinois University Edwardsville 
Southern Utah University 
Southwest Baptist University 
Southwest Missouri State University 
Southwest State University 




37 



Special Study Survey 36 



Southwestern Oklahoma State University 

Spalding University 

Spring Arbor College 

State University of New York at Oswego 

State University of New York, College at 

State University of West Georgia 

Stetson University 

Stillman College 

Suffolk University 

Taylor University 

Tennessee Technological University 

Texas Wesleyan University 

Texas Woman's University 

The College of New Rochelle 

Transylvania University 

Tri-State University 

Troy State University 

Troy State University Dothan 

Tuskegee University 

Union College 

Union University 

University of Akron 

University of Arizona 

University of Arkansas 

University of Arkansas at Little Rock 

University of Arkansas at Monticello 

University of Central Arkansas 

University of Central Florida 

University of Central Oklahoma 

University of Cincinnati 

University of Colorado at Boulder 

University of Colorado at Denver 

University of Connecticut 

University of Evansville 

University of Florida 

University of Great Falls 

University of Hawaii at Manoa 

University of Houston-Clear Lake 

University of Iowa 

University of Kansas 

University of Louisiana at Lafayette 

University of Louisiana at Monroe 

University of Louisville 

University of Maine at Farmington 

University of Maryland, Baltimore County 

University of Massachusetts Lowell 

University of Memphis 

University of Minnesota 

University of Minnesota, Duluth 

University of Missouri- Columbia 
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University of Missouri-Saint Louis 

University of Mobile 

University of Montana 

University of Montevallo 

University of Nebraska at Omaha 

University of Nevada, Reno 

University of New Mexico 

University of North Alabama 

University of North Carolina at Asheville 

University of North Carolina at Chapel Hill 

University of North Carolina at Wilmington 

University of North Dakota 

University of North Florida 

University of North Texas 

University of Oregon 

University of Phoenix 

University of Portland 
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